GraphENS: Neighbor-Aware Ego Network Synthesis for Class-Imbalanced Node Classification

1 INTRODUCTION . Node classification for graphs has attracted significant attention as the importance of large-scale graphs analysis increases in various domains such as bioinformatics and commercial graphs to name a few ( Perozzi et al. , 2016 ; Hamilton et al. , 2017 ; Ying et al. , 2018 ; Mohammadrezaei et al. , 2018 ) . For example , in retail services , acquiring the qualitative node representations for items or customers is critical for improving the quality of recommendation systems ( Perozzi et al. , 2016 ; Ying et al. , 2018 ) . Detecting abnormal users in social networks , as another example , is also closely related to classifying the property of each node ( Mohammadrezaei et al. , 2018 ) . Recently , Graph Neural Networks ( GNNs ) have demonstrated their effectiveness on learning node representations ( Hamilton et al. , 2017 ; Kipf & Welling , 2017 ; Velickovic et al. ) . However , the nodes in many real-world graphs are sometimes class-imbalanced ( Mohammadrezaei et al. , 2018 ; Wang et al. , 2020a ) , hence GNNs are prone to be biased toward major classes , as in general class-imbalance tasks . This bias forces networks to poorly classify the nodes of minor classes , resulting in destructive impacts and a large cost to their services . While the peculiar characteristics of imbalanced node classification and specialized solutions suitable for it have hardly been investigated , applying generic imbalance handling methods ( Chawla et al. , 2002 ; Cui et al. , 2019 ; Cao et al. , 2019 ; Kang et al. , 2020 ; Menon et al. , 2021 ) directly to the graph domain has several non-trivial challenges . One of the distinct natures of graph data is that adjacent nodes are involved in constructing the representation of each node , which makes the model more confused to learn the unbiased representation of minor class nodes . Here , we hypothesize that it is in fact more serious to overfitting to neighbors of minor nodes than to overfitting to the node feature itself . This ‘ neighbor memorization ’ is the critical obstacle to naively adopt the class-imbalance approaches of other domains such as re-weighting and re-sampling used in image classification . Specifically , re-weighting approaches ( Cui et al. , 2019 ; Tan et al. , 2020 ; Cao et al. , 2019 ; Menon et al. , 2021 ; Hong et al. , 2021 ) , applying penalties according to the number of data , simply assign large weight to minor nodes , hence there is no change in neighbor sets of minor nodes observed during training . Re-sampling methods ( Chawla et al. , 2002 ; Kang et al. , 2020 ; Zhang et al. , 2021 ; Wang et al. , 2021 ) , sampling data to balance the number of data for each class , are also vulner- able to overfit on minor nodes with their neighbors . Another challenge of the re-sampling method , especially for oversampling variants , is determining how to connect the newly sampled nodes to the original graph . For example , simply connecting an oversampled node with all neighbors of original node will change the edges of neighbor nodes as well and hence significantly alters the message passing to the neighbors , which might impair their class semantics . To mitigate this issue , GraphSMOTE ( Zhao et al. , 2021 ) exploits edge predictor to decide the connectivity with the neighbors of two minor nodes used in oversampling . Nevertheless , GNNs trained with GraphSMOTE still suffer from neighbor memorization when the number of minor nodes is limited . In this paper , we propose GraphENS , a novel augmentation approach that synthesizes the whole ego network for minor class ( minor node and its one-hop neighbors ) by interpolating two different ego networks in the data ; to enlarge the limited neighbor views of minor instances in the data , our method combines the ego network of anchoring minor node with that of randomly selected node from all classes , where it interpolates ego networks based on KL-divergence between model predictions of ego networks in order to keep the semantics of minor classes . Synthesized ego networks are attached to the original graph to construct a class-balanced graph , and GNNs are trained with the enlarged graph . GraphENS enables the model to learn the minor class nodes with feasible neighbors by generating the virtual ego networks . To further prevent the synthesis of deleterious ego networks , we introduce a saliency-based node mixing approach to generate the central node of ego network . Our method separates class-generic node features from class-specific node features by using saliency information of each feature , and exploits only class-generic attributes to combine with node feature of anchoring minor node . We validate our method on various benchmark datasets including citation networks ( Sen et al. , 2008 ) , and Amazon product co-purchasing networks ( Shchur et al. , 2018 ) with diverse architectures such as GCN ( Kipf & Welling , 2017 ) , GAT ( Velickovic et al . ) , and GraphSAGE ( Hamilton et al. , 2017 ) , and confirm that our approach consistently outperforms baseline methods over various settings . In summary , our contribution is threefold : • We demonstrate that in class-imbalanced node classification , GNNs severely overfit to neighbor sets of minor class nodes , rather than to minor nodes themselves . This ‘ neighbor memorization ’ problem becomes severe when the number of minor nodes is extremely small . • Our method effectively alleviates the neighbor memorization problem in class-imbalanced graphs by synthesizing feasible ego networks based on the similarity between source ego networks . We also block the injection of harmful features in generating the mixed nodes using node feature saliency . • Through extensive experiments , we show that our approach consistently outperforms baselines on multiple benchmark datasets including real-world imbalanced datasets . Even in highly imbalanced synthetic graphs , our method exhibits superior performance . 2 RELATED WORK AND PRELIMINARY . 2.1 CLASS IMBALANCE PROBLEM . The goal of class-imbalance handling in classification is to construct an unbiased classifier to the label distribution of the training set . There are three main streams : loss modification , post-hoc correction , and re-sampling approaches . Loss modification methods alter the objective function to assign more weights ( Japkowicz & Stephen , 2002 ; Cui et al. , 2019 ) or margins ( Tan et al. , 2020 ; Cao et al. , 2019 ; Menon et al. , 2021 ) on minor classes . Post-hoc correction strategies ( Kang et al. , 2020 ; Tian et al. , 2020 ; Menon et al. , 2021 ; Hong et al. , 2021 ) remedy logits to compensate minor classes in the inference . Re-sampling approaches augment minor class data by sampling strategies ( Kang et al. , 2020 ; Liu et al. , 2020 ; Ren et al. , 2020 ) or generation ( Chawla et al. , 2002 ; Kim et al. , 2020a ; Chu et al. , 2020 ; Zhang et al. , 2021 ; Wang et al. , 2021 ) . Among minor class generation approaches , SMOTE ( Chawla et al. , 2002 ) is a widely used method to mix minor data with the nearest data of the identical class . Synthesizing minor class data from data of other classes ( Kim et al. , 2020a ; Chu et al. , 2020 ; Wang et al. , 2021 ) is introduced to exploit the rich information of other classes . Kim et al . ( 2020a ) produces new minor class data by taking gradient steps to translate major class data into minor class data . Wang et al . ( 2021 ) synthesizes minor class data by combining features of minor class data with feature displacements of other data . To extend these approaches to the graph domain , structural aspects of graph have to be considered when generating minor instances . In node classification , imbalance handling works ( Zhou et al. , 2018 ; Wang et al. , 2020b ; Shi et al. , 2020 ; Zhao et al. , 2021 ; Qu et al. , 2021 ) are proposed to exploit structural information in graphs . DR-GCN ( Shi et al. , 2020 ) produces the virtual minor nodes generated by additional conditional GAN and regularizes the features of virtual nodes close to adjacent nodes . GraphSMOTE ( Zhao et al. , 2021 ) generates synthetic minor nodes by interpolating two minor class nodes and a ( pretrained ) edge predictor determines the connectivity of synthetized nodes between synthetized nodes and neighbors of two source minor nodes . ImGAGN ( Qu et al. , 2021 ) synthesizes minor nodes by interpolating features among whole minor nodes with the generated weight matrix . Then the synthesized nodes are connected to the original minor nodes if weights in the matrix are larger than a fixed threshold . As GraphSMOTE and ImGAGN only utilize nodes of the identical minor class to generate minor nodes , the sample diversity of synthesized nodes would be significantly constrained . Moreover , ImGAGN mainly targets binary classification and its extension to multi-class classification is non-trivial since an independent generator is required per each class . Compared to these approaches , our GraphENS utilizes whole nodes to synthesize minor nodes , thus our method outperforms baselines when the number of minor classes is low in Section 5.3 ( Table 4 ) . 2.2 GRAPH NEURAL NETWORKS FOR NODE CLASSIFICATION . We briefly introduce GNNs for node classification tasks . Let us first define graph G ( V , E ) where V is the set of nodes and E is the set of edges between two nodes . Let X ∈ R|V |×d be the node features whose the i-th row represents the d-dimensional feature of the i-th node . N ( v ) is the set of adjacent nodes that are directly connected to v : { u ∈ V | { u , v } ∈ E } . In node classification , each node in the graph corresponds to a class y ∈ { 1 , . . . , C } where C is the number of classes . In this paper , we consider several variants of GNNs that consists of three differentiable functions : 1 ) message function ml , 2 ) permutation invariant message aggregation function φl , and 3 ) node update function hl . Let x ( l ) v be the latent vector of node v at layer l. To simplify notations for the recursive definition of GNNs , we use x ( 0 ) v to denote the input node feature . At each GNN layer , node features are updated as x ( l+1 ) v = hl ( x ( l ) v , φl ( { ml ( x ( l ) v , x ( l ) u , ev , u ) |u ∈ N ( v ) } ) to consider adjacent node embeddings . By passing these aggregated node features to a linear classifier as input , we obtain the prediction ŷ = f ( xv ) for node v where ŷc = P ( y = c|xv ) . As a representative example , a layer of Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) is defined as x ( l+1 ) v = Θ ∑ u∈N ( v ) ∪ { v } ev , u√ d̂ud̂v x ( l ) u where d̂v = 1 + ∑ u∈N ( v ) ev , u with ev , u is the edge weight of edge { u , v } ∈ E and Θ is a filter parameters . There are several variants depending on how these key components are designed , but they are beyond scope of our work .

Summary: This paper addresses the class imbalance problem for node classification in a graph and points out some issues faced by existing GNNs and  methods to address the same. It nicely depicts the problem of overfitting to minor classes and neighborhood memorization problem for class imbalance through experiments. It proposes n approach GraphENS which can work with any message passing GNN to resolve the problem of class imbalance for node classification. Experimental results show the merit of the proposed algorithm working with multiple GNNs (GCN, GraphSAGE and GAT) on both synthetically imbalanced dataset and real-world imbalanced dataset for node classification.