Topology-Imbalance Learning for Semi-Supervised Node Classification

Abstract: The class imbalance problem, as an important issue in learning node representations, has drawn increasing attention from the community. Although the imbalance considered by existing studies roots from the unequal quantity of labeled examples in different classes (quantity imbalance), we argue that graph data expose a unique source of imbalance from the asymmetric topological properties of the labeled nodes, i.e., labeled nodes are not equal in terms of their structural role in the graph (topology imbalance). In this work, we first probe the previously unknown topology-imbalance issue, including its characteristics, causes, and threats to semisupervised node classification learning. We then provide a unified view to jointly analyzing the quantity- and topology- imbalance issues by considering the node influence shift phenomenon with the Label Propagation algorithm. In light of our analysis, we devise an influence conflict detection–based metric Totoro to measure the degree of graph topology imbalance and propose a model-agnostic method ReNode to address the topology-imbalance issue by re-weighting the influence of labeled nodes adaptively based on their relative positions to class boundaries. Systematic experiments demonstrate the effectiveness and generalizability of our method in relieving topology-imbalance issue and promoting semi-supervised node classification. The further analysis unveils varied sensitivity of different graph neural networks (GNNs) to topology imbalance, which may serve as a new perspective in evaluating GNN architectures.


Reviews and author responses:

 Thank you for your clarification. The response clarify part of my concerns. However, the 'analysis' for loss function  provided by the authors remain heuristics, lacking a systematic and convincing explanation. An advanced theoretical analysis may improve the overall quality of this work. I do like the experiment part of this work thus I will keep my rate.

 Thank you for your insightful response! Here is our reply to the further concern:

[Q1] Response to supplementary question about Figure 2(a):

Yes, it is true that the more blue and red nodes, the more similar the prediction results from GCN and LP.

\
[Q2] Response to "the effectiveness of your method highly relies on the distribution of labeled nodes":

It is quite true that the randomly sampled training sets would bring various degrees of topology imbalance. While our ranking-based re-weighting strategy can tell **which nodes are better than others** and make them play more important roles in training to relieve the topology imbalance for training sets with different topology-imbalance degrees (shown in Table2).  

As for "if all the nodes from class B are around the decision boundary and will get similar training weights", the ranking strategy and the cosine mapping function in our method can still distinguish different labeling nodes. Moreover, due to the sparsity of labeled nodes and the randomness of the sampling strategy,  we think the probability is small that all the labeled nodes are close to the boundary and have symmetrical topology.

\
[Q3] Response to "utilizing those labeled nodes close to the boundary by re-designing the graph filter can lead to more elegant and solid solutions":

Our conflict-based re-ranking method and the potential graph filter solution are **orthogonal directions** for the important but ignored topology imbalance issue that can be applied jointly. 
As the first study on the challenging topology imbalance issue, we aim to provide a lite and effective solution for general graph application scenarios (TINL, QINL, and large-scale-graphs with multiple datasets and GNNs), and we hope it can serve as a reliable reference for future studies, including the graph filter solution. 


[Q4] Response to "training a GCN model at first and then re-training another GCN model reweighted by the results from the first GCN model may also work":

I am not quite sure about the meaning of "re-training another GCN model reweighted by the results from the first GCN". I understand it means re-weighting the training nodes by their prediction confidence. Please inform me if there is any misunderstanding.

This self-learning-like solution would be largely affected by the topology imbalance in the first round of training: the decision boundary has already shifted and the prediction result is biased, thus the re-training based on the biased result can hardly correct itself. Besides, it will cost more computing resources to train a model for several rounds. 

Our method can relieve topology imbalance at the beginning of training and can be finished in only one round. The calculation cost of the metric Totoro is very limited even for the enormously large-scale graphs as shown in Section 3.4.








 Thanks for your clarification.

From your response to Q1, if there are more blue and red nodes, the more similar the prediction results from GCN and LP are, right? If so, from the prediction results' perspective, they do have similar decision boundary. 

From your response to Q2, your explanation makes sense. Take a 2-way classification scenario as example. It is true that if labelled nodes from class A are all far from the decision boundary and labelled nodes from class B are all close to the decision boundary, then the decision boundary will be pushed towards the class A. However, here are two concerns: (1) the effectiveness of your method highly relies on the distribution of labelled nodes; Seems that your method cannot handle the above scenario if all the nodes from class B are around the decision boundary and will get similar training weights, which are useless for the calibration of decision boundary. (2) I believe that the key point of this problem is non-parameterized convolution (propagation & aggregation) operation. Models like GCN and LP directly use a **fixed** filter (i.e., symmetrically normalized A or A with self-loop) to do the propagation and aggregation, which leads to the significantly shift of decision boundary as you pointed out. However, such shift may be mitigated if the GNN model contains adaptive aggregator or can incorporate information beyond low-pass filter (e.g., GAT [1] and FAGCN [2]). Intuitively models above-mentioned will not smooth the decision boundary by default but aggregate less information across the boundary (due to the difference of representations across the boundary) and aggregate more information from the same side of the boundary. In other words, they may strengthen the decision boundary through labelled nodes close to the boundary.

Based on my above concern, the topology imbalance problem itself is very interesting. However, simply down-weighting the labelled nodes close to the boundary is far from good enough. Utilizing those labelled nodes close to the boundary by re-designing the graph filter can lead to more elegant and solid solutions.

BTW, whether we really need to locate the labelled nodes far from the boundary based on the results of LP? Training a GCN model at first and then re-training another GCN model reweighted by the results from the first GCN model may also work. This simple solution shares the similar idea as your method except that we do not base on LP and a new metric Totoro.

[1] Veličković, Petar, et al. "Graph Attention Networks." International Conference on Learning Representations. 2018.
[2] Bo, Deyu, et al. "Beyond Low-frequency Information in Graph Convolutional Networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 5. 2021.

 Thanks for your valuable feedback! Here is our response to the concerned questions.

[Q1] Response to question about Figure 2(a):

Sorry for this figure make you confused. GCN/LP=T(rue) means the prediction result of GCN/Label Propagation is correct, while GCN/LP=F(alse) means the prediction result of GCN/Label Propagation is wrong. We can find from Figure 2(a) that there is a strong consistency between the prediction results of GCN and LP. Besides, existing work [1] has also proven that LP can be unified with the GCN model theoretically.

[1] Unifying Graph Convolutional Neural Networks and Label Propagation.

\
[Q2] Response to question about selecting the 'easy' nodes:

It is a good question that why we assign more weights to those 'easy' nodes instead of the 'hard' nodes. Different from the SVM model which is mainly used in supervised learning and relies on the 'hard' samples (support vector) to determine the classification boundaries, most message-passing based GNN model (e.g. GCN) applied in the semi-supervised node classification task relies on smoothing the adjacent nodes to transfer the category information from the labeled nodes to the unlabeled nodes [2]. Thus, the 'easy' labeled nodes far away from the class boundaries are expected to better represent class prototypes. Enlarging the training weights of those 'hard' nodes that are close to the class boundaries makes it easier to confuse the class prototype with others. Besides, the labeling size in semi-supervised learning is much smaller than supervised learning (usually 20 nodes per class) and usually, the training nodes are sampled randomly. Hence, a very likely scenario is that the 'hard' samples for some categories are very close to the true class boundaries, while the 'hard' samples for other categories are far away from the true class boundaries. Relying on these 'hard' nodes to decide decision boundaries will cause a large shift of decision boundaries from the true ones. In fact, the labeled nodes close to the boundary of the class edge are the direct cause of topology imbalance according to our analysis in Section 2.

[2] Deeper Insights into Graph Convolutional Networks for Semi-supervised Learning.

\
[Q3] Response to question about the weight schedule mechanism:

Apart from the proposed loss function, we also try some other weight schedule methods as shown in Appendix B. Overall, our ReNode method has the following advantages: (1) Modifying the training weights of each node enables our method to be combined with most existing GNN encoders and loss functions in a plug-and-play manner and makes it is to follow and extend. Table 1 and Table 3 demonstrate the generalizability of our method; (2) Using the sorting orders instead of the original Totoro values makes the training weight distribution more robust and less susceptible to the outlier nodes; another advantage of sorting strategy lies in that: we can hardly know which nodes are more likely to cause topology imbalance, but we can tell which nodes are better than others and then we can handle different randomly sampled training sets with various degree of topology imbalance; (3) Comparing to the linear decay schedule, the cosine schedule methods pay more attention to nodes with middle-level conflict, distinguishing which is of great importance for reducing topology imbalance. You can refer to Appendix B for more details if you have any interest.

\
[Q4] Response to the advice to the paper writing:

Thanks for these helpful suggestions, and we will refine them in the revised paper.

 Thanks for your insightful comments on our work! Here is our response to the concerned questions.

[Q1] Response to whether there are existing similar studies.

We conducted a detailed literature survey before the start of this work. The survey directions include Lable Propagation, GNN, general imbalance learning, and graph-specific imbalance learning. Our work is the first to point out the important topology imbalance problem, and also proposes the first solution for this challenging issue.

\
[Q2]Response to the design of the loss function.

Apart from the proposed loss function, we also try some other weight schedule methods as shown in Appendix B. Overall, our ReNode method has the following advantages: (1) Modifying the training weights of each node enables our method to be combined with most existing GNN encoders and loss functions in a plug-and-play manner and makes it is to follow and extend. Table 1 and Table 3 demonstrate the generalizability of our method; (2) Using the sorting orders instead of the original Totoro values makes the training weight distribution more robust and less susceptible to the outlier nodes; another advantage of sorting strategy lies in that: we can hardly know which nodes are more likely to cause topology imbalance, but we can tell which nodes are better than others and then we can handle different randomly sampled training sets with various degree of topology imbalance; (3) Comparing to the linear decay schedule, the cosine schedule methods pay more attention to nodes with middle-level conflict, distinguishing which is of great importance for reducing topology imbalance. You can refer to Appendix B for more details if you have any interest. Inevitably, calculating training weights will bring additional computational costs. However, due to the limited size of training nodes, the training weights can be calculated quickly before training. We successfully apply our method in the enormous MAG-Scholar dataset which has more than 10 million nodes as shown in Section 3.4, and the increase in training time is less than 3%. Moreover, there is no method proposed before for the topology-imbalance issue, and our method provides the first solution for the difficult and neglected topology-imbalance issue. The effectiveness of our method is verified in various semi-supervised node classification scenes: TINL-only, TINL & QINL, Large-scale graph, which demonstrates that the topology-imbalance issue is ubiquitous in GNN training, and solving the topology-imbalance issue is a promising direction for improving the related graph tasks.

\
[Q3] Response to whether there exist other topology imbalance baselines.

As stated in the response to Q1, we did not find any methods proposed for solving the topology imbalance issue after a detailed literature survey. Comparing to imbalance learning for general scenes, there are fewer studies related to graph imbalance learning, and all of them focus on the quantity-imbalance issue. We implemented three recently proposed graph quantity-imbalance methods, which encode the topological information and may be helpful for the topology-imbalance issue, but the results in Table 3 show that our proposed method consistently outperforms these three baselines across multiple graph datasets, demonstrating the effectiveness of our ReNode method.

 Thanks for your detailed comments! Here is our response to the concerned questions.

[Q1] Response to the homophilous assumption.

The homogeneously connected graphs have numerous application scenarios (such as citation networks, social media, E-shopping，, etc.) and have attracted enormous studies. Moreover, many extremely influential graph-related studies are based on the research on homogeneously connected graphs (such as GCN, GAT, GraphSAGE, PPNP, etc.). In OGB [1], the authors build a new graph benchmark for graph models. For the node classification task, 4 out of the 5 proposed datasets are homogeneously connected, telling the importance and representativeness of the homogeneously connected graphs in the node classification study. Besides, the main intention of this work is to point out the problem of Topology Imbalance, a highly ubiquitous, pernicious but ignored issue in graph scene. In this work, we take the most fundamental semi-supervised node classification scenario with the homophilous assumption as an example to study the topology imbalance. We believe the topology imbalance issue is widespread in other graph tasks with different patterns; how to measure and solve the topology-imbalance issues in broader graph scopes remains a meaningful challenge for the future and the conclusion in this work may inspire future studies.

[1] Open Graph Benchmark: Datasets for Machine Learning on Graphs. (Neurips2020)

\
[Q2] Response to the improvement gap of ReNode.

Our ReNode method is proposed for handling the topology imbalance issue, which is orthogonal to the existing methods like CB or re-weighting focusing on the quantity imbalance issue. As shown in Table1, even when the training set is quantity-balanced, our ReNode methods can still improve the results of multiple GNN models and graph datasets by relieving topology imbalance. Moreover, as shown in Table 3, our method can be combined with the quantity-imbalance methods to further solve the comprehensive graph imbalance issue. Besides, the random dataset partitioning will bring varying degrees of topology imbalance to the training set. From Table 2, we can find that our method brings larger improvements (absolute>2%) when the topology imbalance level is high.

\
[Q3] Response to the lack of the baseline considering TINL and QINL at the same time.

We conducted a detailed literature survey before the start of this work. The survey directions include Lable Propagation, GNN, general imbalance learning, and graph-specific imbalance learning. Our work is the first to point out the important topology imbalance problem, and also proposes the first solution for this challenging issue. Thus, there are no available baselines considering TINL and QINL at the same time. The effectiveness of our method is verified in various semi-supervised node classification scenes: TINL-only, TINL & QINL, and Large-scale graph, which demonstrates that the topology-imbalance issue is ubiquitous in graph scenes, and solving the topology-imbalance issue is a promising direction for the related graph tasks.

\
[Q4] Response to question about the alternative reweighting strategies:

Apart from the proposed weight schedule method, we also try some other weight schedule methods as shown in Appendix B. Overall, our ReNode method has the following advantages: (1) Modifying the training weights of each node enables our method to be combined with most existing GNN encoders and loss functions in a plug-and-play manner and makes it is to follow and extend. Table 1 and Table 3 demonstrate the generalizability of our method; (2) Using the sorting orders instead of the original Totoro values makes the training weight distribution more robust and less susceptible to the outlier nodes; another advantage of sorting strategy lies in that: we can hardly know which nodes are more likely to cause topology imbalance, but we can tell which nodes are better than others and then we can handle different randomly sampled training sets with various degree of topology imbalance; (3) Comparing to the linear decay schedule, the cosine schedule methods pay more attention to nodes with middle-level conflict, distinguishing which is of great importance for reducing topology imbalance. You can refer to Appendix B for more details if you have any interest.

\
[Q5] Response to how ReNode affects different graph convolutions:

According to [2], any GNN models owning the message aggregation, combine, and readout mechanism are as powerful as Weisfeiler-Lehman (WL) test, including GCN, GAT, GraphSAGE, SGC, and so on. Our method works in the readout stage and re-weights the cross-entropy loss (or other loss) of the labeled data. Therefore, although different graph convolutions vary in their message aggregation and combine stages, our ReNode method can release the topology-imbalance issue in the last readout stage. The result is Table 1 verifies the effectiveness of our method in 6 widely used GNN models across 5 benchmark datasets.

[2] How Powerful are Graph Neural Networks?

\
[Q6] Response to "how ReNode (and baselines) performed as imbalance was more smoothly varied"

We mainly study the combination of TINL and QINL in Table 3. We conclude that when the dataset is severely quantity imbalanced, our method can still effectively alleviate the issue of topology imbalance. Since our work is based on topological imbalance, we did not give more space to the analysis of the quantity imbalanced ratio in the main paper, and we will include this analysis experiment recently. Thanks for this suggestion!

\
[Q7] Response to whether the GAT model is tuned well.

We did carry out a detailed hyperparameters search in the submitted paper. Indeed, for each experiment, the hidden size, dropout probability, learning rate, the lower bound, and the upper bound of the weight schedule are tuned. More details about the hyper-parameters searching can be found in Appendix A. Besides, in our implementation, the GAT model head dimension size is set to (hidden size) / (head number), thus the total model parameter size is similar to other models GCN.

\
[Q8] Response to whether there exists a QINL method that helps TINL indirectly.

So far, all the studies of graph imbalance learning focus on the quantity imbalance issue. We implemented three recently proposed graph quantity-imbalance methods (include GraphSMOTE, which contains an edge generator to conduct re-sampling), which encodes the topological information and may be helpful for the topology-imbalance issue, but the results in Table 3 show that our methods consistently outperform these three methods across multiple graph datasets, demonstrating that these methods proposed for quantity imbalance issue can hardly handle the topology imbalance issue.

The authors concerns the topological imbalance problem in node classification problem. The imbalance caused by  asymmetric topological structure during the semi-supervised learning process. The authors introduces the intuition with classical label propagation problem and extended it to the case of learning GNN. To alleviate the pain of the imbalance issue the author introduced the Node reweighting loss, which puts more effort on learning with nodes that have less distance weighted conflicts. 

  clarity: Overall the paper is well written and easy to follow.


originality: I am not sure if topo imbalance problem proposed by the author is entirely new. Seems natural in Label Propagation type problem.  Please confirm.


significance: Ok


quality: The empirical study seems promising. I appreciate the scientific and transparent experimental section.
However, the approach proposed by the authors seems more like a loss type of heuristic, lacking analysis why it needs to be designed in this form and what is the trade off by applying such loss, etc.  In the empirical study section, it seems that the benchmarks are not convincing enough. The ReNode approach may benefit from comparisons with other stronger benchmarks with potential to solve the topo imbalance problem. (please confirm if does not exist).
 yes

This paper proposes an instance-reweighting scheme for GNNs based on the idea that examples "inside" the class's cluster should be more important than ones which are on the boundary.  A heuristic (Totoro) is proposed for the class weight, based on how many other classes the personalized pagerank propagation goes through.  * **Significance**:  While I like some aspects of the write-up, I find there's some blind spots in the work as it currently stands, which limit its significance:

  *   The proposed method has a strong assumption that the data is homophilous (edges are more likely to link nodes sharing the same label, so clusters of the same label are present in the data).  Should this not hold, all nodes are on the graph's "class boundary" (per the Totoro PPR definition). While true for many academic datasets which are hand-selected for this condition (especially citation networks like Cora, etc), this is just *not* the case for most real GNN datasets.  The authors acknowledge this limitation, but the phenomenon needs to be studied in my opinion.

  *   In both absolute *and* relative terms, the performance gains by applying ReNode for all datasets is actually quite low.  (~1% absolute, ~<2% relative?)  This seems to indicate that ReNode doesn't really address imbalance much better than other proposed approaches (e.g. consider Table 3).  Furthermore it seems like there's no baseline in Table 3 that considers both kinds of imbalance (TINL and QINL)?

  *  There seems to be no study of alternative reweighting strategies to validate the choice of PPR.  There are probably dozens of alternatives possible here -- this choice needs to be defended.

  *  Understanding how ReNode works with other models (than GCN) is necessary to understand its significance.  For example, Figure 5 shows that there may be some difference in sensitivity between methods, but we don't know how ReNode effects different graph convolutions.

* **Clarity**:  The paper is well written and easy to understand.  

* **Quality**:  Generally high, but aside from the issues mentioned in significance, there were other obvious analysis that need to be performed:

  *  A more thorough analysis of imbalance and its effect on performance (only two imbalance ratios [5,10] are used...it'd be nice to understand how ReNode (and baselines) performed as imbalance was more smoothly varied.

  *  There's an assertion that  GAT is the least sensitive to topological imbalance (Figure 5), but quite frankly, I don't know if I believe this -- GAT (as the most parameterized model) is much more capable of overfitting.  Instead, I wonder if the hyper-parameters were not adequately searched for all 3 methods for each imbalance setting.

* **Originality**:  There's a number of work out there which considers quantity and graph imbalance, so the onus is on ReNode to show that its unique.  I'm curious if related work for graphs *really* doesn't consider topology imbalance.  To be more specific, some of the regularizations to tackle quantity imbalance might end up encoding topology indirectly (e.g. via unlabeled nodes, edge generation, or something else).  

  *  There has also been a lot of work studying PPR & GNNs lately, but the authors seem to do an ok job surveying the field. There's a brief discussion about the limitation of the method due to its homophily assumption, but I think this is a big blocker that we need to understand more fully.

Authors study a problem named topology imbalance which is led from the topological difference of labelled nodes in a given graph. It degrades the performance of node classifiers by 'shifting' the decision boundary. The main contributions of this paper are (1) it points out the topology imbalance problem in node classification tasks; (2) it proposes a metric (Totoro) to measure the topology imbalance; (3) it designs an annealing method to reweight samples based on their topological importance; (4) authors provide vast experimental results over multiple scenarios, datasets, and baselines.  The topology imbalance problem is very interesting and should be paid more attention. It is clearly illustrated by the authors that labelled nodes at different positions should be weighted differently. Figure 3 is quite impressive that the proposed metric Totoro has strong indication about the position of the labelled nodes and the metric value is directly related with the performance of downstream node classifiers. Overall the presentation is good and I can follow most ideas presented. Specifically, I have following concerns:
(1) A very important assumption this paper holds is that the decision boundary of Label propagation methods and GNNs methods are aligned. Authors verify this by a visualization in Figure 2(a) but this is very messy and I feel confused about the meaning of the legends (e.g., GCN=T, LP=T).
(2) Authors empirically proves that nodes with large Totoro values are far away from the decision boundary and should be set large weights during training. Experiment results indeed demonstrate the performance gaining. However, it is still non-intuitive. Samples far away from the decision boundary should be easy cases and contribute less to the classification in many models (refer to SVM). Assigning larger weights to the easy samples is not intuitive for me. Also, the weight computed based on Rank of the Totoro metric is quite heuristic. It is fine but it would be better if authors can point out the direct connection between model performance, specific Totoro values and sample weights.
(3) The presentation of the core idea from line 120 to line 122 can be improved.  The operation '*' in Eq. (6) and (7) is multiplication? If so try to replace it with other symbols since '*' represents convolution in many cases. Line 198 'GTA' should be 'GAT'.  I agree with authors mentioned in the checklist that their submission has no obvious potential negative societal effect.

Meta-review: This paper investigates the issue that nodes in a graph may have different local topology. This imbalance of topology may induce bias in node classification task. The paper proposes a solution inspired by classic label propagation. Nodes far away from decision boundary are assigned heavier weights in learning, whereas nodes near decision boundary are considered less important. 

While reviewers generally thought the paper does bring up an important and interesting problem, they did have many concerns that were not addressed after rebuttal. 

1) the problem statement (topology imbalance) lacks a rigorous / formal definition. This makes it harder to appreciate the proposed loss. 
2) the assumption (homophilous) may be too strong.
3) the proposed approach is too simplistic: nodes near the decision boundary should be used smartly, not be discarded.





Decision: accept