Topology-Imbalance Learning for Semi-Supervised Node Classification

1 Introduction . Graph is a widely-used data structure [ 51 ] , where the nodes are connected to each other through natural or handcrafted edges . Similar to other data structures , the representation learning for node classification faces the challenge of quantity-imbalance issue , where the labeling size varies among classes and the decision boundaries of trained classifiers are mainly decided by the majority classes [ 46 ] . There have been a series of studies [ 35 , 11 , 49 ] handling the Quantity-Imbalance Node Representation Learning ( short as QINL ) . However , different with other data structures , graph-structured data suffers from another aspect of the imbalance problem : the imbalance caused by the asymmetric and uneven topology of labeled nodes , where the decision boundaries are driven by the labeled nodes close to the topological class boundaries ( left of Figure 1 ) thus interfering with the model learning . Present Work . For the first time , we recognize the Topology-Imbalance Node Representation Learning ( short as TINL ) as a graph-specific imbalance learning topic , which mainly focus on the 1The code is available at https : //github.com/victorchen96/ReNode . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . decision boundaries shift phenomena driven by the topology imbalance in graph and is an essential component for node imbalance learning . Comparing with the well-explored QINL that studies the imbalance caused by the numbers of labeled nodes , TINL explores the imbalance caused by the positions of labeled nodes and owns the following characteristics : • Ubiquity : Due to the complex connections of the graph nodes , the topology structure of nodes in different categories is naturally asymmetric , which makes TINL an essential characteristic in node representation learning . Hence , it is difficult to construct a completely symmetric labeling set even with an abundant annotation budget . • Perniciousness : The influence from labeled nodes decays with the topology distance [ 3 ] . The asymmetric topology of labeled nodes in different classes and the uneven distribution of labeled nodes in the same class will cause the influence conflict and influence insufficient problems ( left of Figure 1 ) respectively , resulting in a shift of decision boundaries . • Orthogonality : Quantity-imbalance studies [ 49 , 8 , 5 ] usually treat the labeled nodes of the same class as a whole and devise solutions based on the total numbers of each class , while TINL explores the influence of the unique position of each labeled node on decision boundaries . Thus , TINL is independent of QINL in terms of the object of study . Exploring TINL is of great importance for node representation learning due to its ubiquity and perniciousness . However , the methods [ 17 , 22 ] for quantity imbalance can be hardly applied to TINL because of the orthogonality . To remedy the topology-imbalance issue , thus promoting the node classification , we propose a model-agnostic training framework ReNode to re-weight the labeled nodes according to their positions . We devise the conflict detection-based Topology Relative Location ( Totoro ) metric to leverage the interaction among labeled nodes across the whole graph to locate their structural positions . Based on the Totoro metric , we further increase the training weights of nodes with small conflict that are highly likely to be close to topological class centers to make them play a more pivotal role during training , and vice versa ( right of Figure 1 ) . Empirical results of various imbalance scenarios ( TINL , QINL , large-scale graph ) and multiple graph neural networks ( GNNs ) demonstrate the effectiveness and generalizability of our method . Besides , we provide the sensitivity to topology imbalance as a new evaluation perspective for different GNN architectures . 2 Topology-Imbalance Node Representation Learning . 2.1 Notations and Preliminary . In this work , we follow the well-established semi-supervised node classification setting [ 47 , 18 ] to conduct analyses and experiments . Given an undirected and unweighted graph G = ( V , E , L ) , where V is the node set represented by the feature matrixX ∈ Rn∗d ( n = |V | is the node size and d is the node embedding dimension ) , E is the edge set which is represented by an adjacency matrix A ∈ Rn∗n , L ⊂ V is the labeled node set and usually we have |L| |V | , the node classification task is to train a classifier F ( usually a GNN ) to predict the class label y for the unlabeled node set U = V −L . The training sets for different classes are represented by ( C1 , C2 , · · · , Ck ) and k is the number of classes . The labeling ratio δ = L/V is the proportion of labeled nodes in all nodes . In this work , we focus on TINL in homogeneously-connected graphs and hope to inspire future studies on the critical topology-imbalance issue . 2.2 Understanding Topology Imbalance via Label Propagation . From Figure 1 , we can intuitively perceive the imbalance brought by the positions of labeled nodes ; in this part , we further explore the nature of topology imbalance with the well-known Label Propagation [ 50 ] algorithm ( short as LP ) and provide a uniform analysis framework for the comprehensive node imbalance issue . In LP , labels are propagated from the labeled nodes and aggregated along edges , which can also be viewed as a random walk process from labeled nodes . The convergence result Y after repeated propagation is regarded as the nodes soft-labels : Y = α ( I − ( 1− α ) A′ ) −1Y 0 , ( 1 ) where I is the identity matrix , α ∈ ( 0 , 1 ] is the random walk restart probability , A′ = D− 12AD− 12 is the adjacency matrix normalized by the diagonal degree matrixD , Y 0 is the initial label distribution where labeled nodes are represented by the one-hot vectors . The prediction label for the i-th node is qi = arg maxj Yij . LP is a simple yet successful model [ 37 ] and can be unified with GNN models owning the message-passing mechanism [ 41 ] . From Figure 2 ( a ) , we can empirically find that there is a significant correlation between the results of LP and GCN ( T/F indicates prediction is True/False ) . The LP prediction q can be viewed as the distribution of the ( labeled ) node influence [ 41 ] ( i.e . each node is mostly influenced by which class ’ s information ) ; hence the boundaries of the node influence can act as an effective reflection for the GNN model decision boundaries considering the high consistency between LP and GNN . Moreover , node influence offers a unified view of TINL and QINL : ideally , the node influence boundaries should be consistent with the true class boundaries , but both the labeled nodes ’ numbers ( QINL ) and positions ( TINL ) can cause a shift of the node influence boundaries from the true one , resulting in deviation of the model decision boundaries . Node imbalance issue is composed of topology- and quantity-imbalance . Figure 2 illustrates two examples of node influence boundary shift . In Figure 2 ( b ) , when the uniform selection is adopted to generate training set , both the quantity and the topology are imbalanced for model training ; then the large class with more total nodes ( denotes by blue color ) will own stronger influence than the small class with fewer total nodes ( denotes by red color ) due to the quantity advantage and the node influence boundary is shifted towards the small class . In Figure 2 ( c ) , when the quantity-balanced strategy is adopted for sampling training nodes , it will be easier for the small class to has more labeled nodes close to the class boundary and the boundary of the node influence is shifted into the large class . We can find that even when the training set is quantity-balanced , the topology-imbalance issue still exists and hinders the node classification learning . Hence , we can conclude that node imbalance learning is caused by the joint effect of TINL and QINL . Separately considering TINL or QINL will lead to a one-sided solution to node imbalance learning . 2.3 Measuring Topology Imbalance by Influence Conflict . Although we have realized that the imbalance of node topology interferes with model learning , how to measure the labeled node ’ s relative topological position to its class ( being far away from or close to the class center ) remains the key challenge in handling the topology-imbalance issue due to the complex graph connections and the unknown class labels for most nodes in the graph . As the nodes are homogeneously connected when constructing the graph , even nodes close to the class boundaries own similar characteristics to their neighbors . Thus it is unreliable to leverage the difference between the characteristics of one labeled node and its surrounding subgraphs to locate its topological position . Instead , we propose to utilize the node topology information by considering the node influence conflict across the whole graph and devise the Conflict Detection-based Topology Relative Location metric ( Totoro ) . Similar to Eq ( 1 ) , we calculate the Personalized PageRank [ 27 ] matrix P to measure node influence distribution from each labeled node : P = α ( I − ( 1− α ) A′ ) −1 . ( 2 ) Node influence conflict denotes topological position . According to related studies [ 41 , 19 , 2 ] , P can be viewed as the distribution of influence exerted outward from each node . We assume that if a labeled node v ∈ V encounters strong heterogeneous influence from the other classes ’ labeled nodes in the subgraph around node v where node v itself owns great influence , we have the conclusion that node v meets large influence conflict in message passing and it is close to topological class boundaries , and vice versa . Based on this hypothesis , we take the expectation of the influence conflict between the node v and the labeled nodes from other classes when node v randomly walks across the entire graph as a measurement of how topologically close node v is to the center of the class it belongs to . The Totoro value of node v is computed as : Tv = Ex∼Pv , : [ ∑ j∈ [ 1 , k ] , j 6=yv 1 |Cj | ∑ i∈Cj Pi , x ] , ( 3 ) where yv is the ground-truth label of node v , Pv indicates the personalized PageRank probability vector for the node v. A larger Totoro value Tv indicates that node v is topologically closer to class boundaries , and vice versa . The normalization item 1/|Cj | is added to make the influence from the different classes comparable when computing conflict . We visualize the node labels and the Totoro values ( scaled to [ 0 , 1 ] ) of labeled nodes in Figure 3 ( a ) . We can find that the labeled nodes with smaller Totoro values are farther away from the class boundaries , demonstrating the effectiveness of Totoro in locating the positions of labeled nodes . Besides , we sum the conflict of all the labeled nodes ∑ b∈L Tv to measure the overall conflict of the dataset , which can be viewed as the metric for the overall topology imbalance given the graph G and the training set L. Figure 3 ( b ) shows that there is a significant negative correlation between the overall conflict and the model performance , which further demonstrates the effectiveness of Totoro in measuring the intensity of topology imbalance at the dataset level .

Summary:  The issue of imbalance is a very critical problem in ML, especially in a realistic setting. In general, imbalance is quantitative, but this paper points out topological imbalance as a graph-specific problem. As a solution to topological imbalance, a method of reweighting called ReNode is proposed and its effectiveness is verified.