Topology-Imbalance Learning for Semi-Supervised Node Classification

1 Introduction . Graph is a widely-used data structure [ 51 ] , where the nodes are connected to each other through natural or handcrafted edges . Similar to other data structures , the representation learning for node classification faces the challenge of quantity-imbalance issue , where the labeling size varies among classes and the decision boundaries of trained classifiers are mainly decided by the majority classes [ 46 ] . There have been a series of studies [ 35 , 11 , 49 ] handling the Quantity-Imbalance Node Representation Learning ( short as QINL ) . However , different with other data structures , graph-structured data suffers from another aspect of the imbalance problem : the imbalance caused by the asymmetric and uneven topology of labeled nodes , where the decision boundaries are driven by the labeled nodes close to the topological class boundaries ( left of Figure 1 ) thus interfering with the model learning . Present Work . For the first time , we recognize the Topology-Imbalance Node Representation Learning ( short as TINL ) as a graph-specific imbalance learning topic , which mainly focus on the 1The code is available at https : //github.com/victorchen96/ReNode . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . decision boundaries shift phenomena driven by the topology imbalance in graph and is an essential component for node imbalance learning . Comparing with the well-explored QINL that studies the imbalance caused by the numbers of labeled nodes , TINL explores the imbalance caused by the positions of labeled nodes and owns the following characteristics : • Ubiquity : Due to the complex connections of the graph nodes , the topology structure of nodes in different categories is naturally asymmetric , which makes TINL an essential characteristic in node representation learning . Hence , it is difficult to construct a completely symmetric labeling set even with an abundant annotation budget . • Perniciousness : The influence from labeled nodes decays with the topology distance [ 3 ] . The asymmetric topology of labeled nodes in different classes and the uneven distribution of labeled nodes in the same class will cause the influence conflict and influence insufficient problems ( left of Figure 1 ) respectively , resulting in a shift of decision boundaries . • Orthogonality : Quantity-imbalance studies [ 49 , 8 , 5 ] usually treat the labeled nodes of the same class as a whole and devise solutions based on the total numbers of each class , while TINL explores the influence of the unique position of each labeled node on decision boundaries . Thus , TINL is independent of QINL in terms of the object of study . Exploring TINL is of great importance for node representation learning due to its ubiquity and perniciousness . However , the methods [ 17 , 22 ] for quantity imbalance can be hardly applied to TINL because of the orthogonality . To remedy the topology-imbalance issue , thus promoting the node classification , we propose a model-agnostic training framework ReNode to re-weight the labeled nodes according to their positions . We devise the conflict detection-based Topology Relative Location ( Totoro ) metric to leverage the interaction among labeled nodes across the whole graph to locate their structural positions . Based on the Totoro metric , we further increase the training weights of nodes with small conflict that are highly likely to be close to topological class centers to make them play a more pivotal role during training , and vice versa ( right of Figure 1 ) . Empirical results of various imbalance scenarios ( TINL , QINL , large-scale graph ) and multiple graph neural networks ( GNNs ) demonstrate the effectiveness and generalizability of our method . Besides , we provide the sensitivity to topology imbalance as a new evaluation perspective for different GNN architectures . 2 Topology-Imbalance Node Representation Learning . 2.1 Notations and Preliminary . In this work , we follow the well-established semi-supervised node classification setting [ 47 , 18 ] to conduct analyses and experiments . Given an undirected and unweighted graph G = ( V , E , L ) , where V is the node set represented by the feature matrixX ∈ Rn∗d ( n = |V | is the node size and d is the node embedding dimension ) , E is the edge set which is represented by an adjacency matrix A ∈ Rn∗n , L ⊂ V is the labeled node set and usually we have |L| |V | , the node classification task is to train a classifier F ( usually a GNN ) to predict the class label y for the unlabeled node set U = V −L . The training sets for different classes are represented by ( C1 , C2 , · · · , Ck ) and k is the number of classes . The labeling ratio δ = L/V is the proportion of labeled nodes in all nodes . In this work , we focus on TINL in homogeneously-connected graphs and hope to inspire future studies on the critical topology-imbalance issue . 2.2 Understanding Topology Imbalance via Label Propagation . From Figure 1 , we can intuitively perceive the imbalance brought by the positions of labeled nodes ; in this part , we further explore the nature of topology imbalance with the well-known Label Propagation [ 50 ] algorithm ( short as LP ) and provide a uniform analysis framework for the comprehensive node imbalance issue . In LP , labels are propagated from the labeled nodes and aggregated along edges , which can also be viewed as a random walk process from labeled nodes . The convergence result Y after repeated propagation is regarded as the nodes soft-labels : Y = α ( I − ( 1− α ) A′ ) −1Y 0 , ( 1 ) where I is the identity matrix , α ∈ ( 0 , 1 ] is the random walk restart probability , A′ = D− 12AD− 12 is the adjacency matrix normalized by the diagonal degree matrixD , Y 0 is the initial label distribution where labeled nodes are represented by the one-hot vectors . The prediction label for the i-th node is qi = arg maxj Yij . LP is a simple yet successful model [ 37 ] and can be unified with GNN models owning the message-passing mechanism [ 41 ] . From Figure 2 ( a ) , we can empirically find that there is a significant correlation between the results of LP and GCN ( T/F indicates prediction is True/False ) . The LP prediction q can be viewed as the distribution of the ( labeled ) node influence [ 41 ] ( i.e . each node is mostly influenced by which class ’ s information ) ; hence the boundaries of the node influence can act as an effective reflection for the GNN model decision boundaries considering the high consistency between LP and GNN . Moreover , node influence offers a unified view of TINL and QINL : ideally , the node influence boundaries should be consistent with the true class boundaries , but both the labeled nodes ’ numbers ( QINL ) and positions ( TINL ) can cause a shift of the node influence boundaries from the true one , resulting in deviation of the model decision boundaries . Node imbalance issue is composed of topology- and quantity-imbalance . Figure 2 illustrates two examples of node influence boundary shift . In Figure 2 ( b ) , when the uniform selection is adopted to generate training set , both the quantity and the topology are imbalanced for model training ; then the large class with more total nodes ( denotes by blue color ) will own stronger influence than the small class with fewer total nodes ( denotes by red color ) due to the quantity advantage and the node influence boundary is shifted towards the small class . In Figure 2 ( c ) , when the quantity-balanced strategy is adopted for sampling training nodes , it will be easier for the small class to has more labeled nodes close to the class boundary and the boundary of the node influence is shifted into the large class . We can find that even when the training set is quantity-balanced , the topology-imbalance issue still exists and hinders the node classification learning . Hence , we can conclude that node imbalance learning is caused by the joint effect of TINL and QINL . Separately considering TINL or QINL will lead to a one-sided solution to node imbalance learning . 2.3 Measuring Topology Imbalance by Influence Conflict . Although we have realized that the imbalance of node topology interferes with model learning , how to measure the labeled node ’ s relative topological position to its class ( being far away from or close to the class center ) remains the key challenge in handling the topology-imbalance issue due to the complex graph connections and the unknown class labels for most nodes in the graph . As the nodes are homogeneously connected when constructing the graph , even nodes close to the class boundaries own similar characteristics to their neighbors . Thus it is unreliable to leverage the difference between the characteristics of one labeled node and its surrounding subgraphs to locate its topological position . Instead , we propose to utilize the node topology information by considering the node influence conflict across the whole graph and devise the Conflict Detection-based Topology Relative Location metric ( Totoro ) . Similar to Eq ( 1 ) , we calculate the Personalized PageRank [ 27 ] matrix P to measure node influence distribution from each labeled node : P = α ( I − ( 1− α ) A′ ) −1 . ( 2 ) Node influence conflict denotes topological position . According to related studies [ 41 , 19 , 2 ] , P can be viewed as the distribution of influence exerted outward from each node . We assume that if a labeled node v ∈ V encounters strong heterogeneous influence from the other classes ’ labeled nodes in the subgraph around node v where node v itself owns great influence , we have the conclusion that node v meets large influence conflict in message passing and it is close to topological class boundaries , and vice versa . Based on this hypothesis , we take the expectation of the influence conflict between the node v and the labeled nodes from other classes when node v randomly walks across the entire graph as a measurement of how topologically close node v is to the center of the class it belongs to . The Totoro value of node v is computed as : Tv = Ex∼Pv , : [ ∑ j∈ [ 1 , k ] , j 6=yv 1 |Cj | ∑ i∈Cj Pi , x ] , ( 3 ) where yv is the ground-truth label of node v , Pv indicates the personalized PageRank probability vector for the node v. A larger Totoro value Tv indicates that node v is topologically closer to class boundaries , and vice versa . The normalization item 1/|Cj | is added to make the influence from the different classes comparable when computing conflict . We visualize the node labels and the Totoro values ( scaled to [ 0 , 1 ] ) of labeled nodes in Figure 3 ( a ) . We can find that the labeled nodes with smaller Totoro values are farther away from the class boundaries , demonstrating the effectiveness of Totoro in locating the positions of labeled nodes . Besides , we sum the conflict of all the labeled nodes ∑ b∈L Tv to measure the overall conflict of the dataset , which can be viewed as the metric for the overall topology imbalance given the graph G and the training set L. Figure 3 ( b ) shows that there is a significant negative correlation between the overall conflict and the model performance , which further demonstrates the effectiveness of Totoro in measuring the intensity of topology imbalance at the dataset level .

Summary: The authors concerns the topological imbalance problem in node classification problem. The imbalance caused by  asymmetric topological structure during the semi-supervised learning process. The authors introduces the intuition with classical label propagation problem and extended it to the case of learning GNN. To alleviate the pain of the imbalance issue the author introduced the Node reweighting loss, which puts more effort on learning with nodes that have less distance weighted conflicts.