PI-GNN: Towards Robust Semi-Supervised Node Classification against Noisy Labels

1 INTRODUCTION . Graphs are ubiquitously used to represent data in different fields , including social networks , bioinformatics , recommendation systems , and computer network security . Accordingly , graph analysis tasks , such as node classification , have a significant impact in reality ( Lan et al. , 2020 ) . The success of machine learning models , such as graph neural networks ( GNNs ) on node classification relies heavily on the collection of large datasets with human-annotated labels ( Zhou et al. , 2019 ) . However , it is extremely expensive and time-consuming to label millions of nodes with high-quality annotations . Therefore , when dealing with large graphs , usually a subset of nodes is labeled , and a wide spectrum of semi-supervised learning techniques have emerged for improving node classification performance ( Zhu et al. , 2003 ; Zhou et al. , 2003 ; Kipf & Welling , 2017 ) . Although achieving promising results , these techniques overlook the existence of noisy node labels . For instance , practitioners often leverage inexpensive alternatives for annotation , such as combining human and machine-generated label ( Hu et al. , 2020 ) , which inevitably yields samples with noisy labels . Since neural networks ( including GNNs ) are able to memorize any given ( random ) labels ( NT et al. , 2019 ; Zhang et al. , 2017 ) , these noisy labels would easily prevent them from generalizing well . Therefore , training robust GNNs for semi-supervised node classification against noisy labels becomes increasingly crucial for safety-critical graph analysis , such as predicting the identity groups of users in social networks or the function of proteins to facilitate wet laboratory experiments , etc . A natural solution is extending the robust training techniques specifically designed for image data into graphs , such as by estimating noise transition matrix ( NT et al. , 2019 ) , re-weighting training samples ( Xia et al. , 2021 ; Li et al. , 2021 ; Dong et al. , 2020 ) and modifying model structure ( Li et al. , 2020a ) , which achieve promising results . However , there are two limitations . Firstly , some techniques require a clean set of node and label pairs for training , which are not practical to obtain under extremely noisy settings ( Li et al. , 2021 ) . Secondly , all of them solely rely on the noisy pointwise input-label interactions during training . Although some of them implicitly exploit other attributes ( the clean graph structure , etc . ) that encodes the pairwise information between nodes , such as aggregating features by GNNs on the input graphs or launching label propagation to get the pseudo labels for unlabeled nodes , they still operate on the noisy embeddings for inference , which may lead to limited performance . Thus , designing a robust GNN training technique for node classification against noisy labels is still a challenging problem . In this paper , we propose a novel framework , which explicitly leverages the Pairwise Interactions in Graph Neural Network ( PI-GNN ) to perform robust training without auxiliary clean labels . The PI , which is based on the similarity of two nodes , is meaningful for GNN against noisy labels because the noise rate for the PI labels can be much lower than that of the pointwise noisy class labels ( Figure 1 ( a ) ) . Consider two nodes from the same class have the same noisy labels , their similarity is still high w.r.t . the PI , which is helpful for the model to learn . Meanwhile , the pointwise information will inject both of these noisy class labels into GNN , which hurts its generalization performance . To ensure an effective PI learning procedure , two important components are highlighted . The first one is how to design informative PI labels . We propose two instantiations : 1 ) we use the adjacency matrix as PI labels so that if two nodes are connected , their PI label is positive . 2 ) We compare the noisy labels , i.e. , two nodes that have the same class labels are given a positive PI label . During training , PI-GNN explicitly forces the embedding similarity of two nodes to be close to the PI labels . The second one is how to deal with the sub-optimal PI labels during regularization . It is critical to notice that the two instantiations are not optimal . Firstly , connected nodes do not guarantee to have the same ( clean ) class label , so their PI label is not always positive . Secondly , the class labels are noisy , and comparing them to obtain the PI labels inevitably introduces noise . To address this problem , we introduce uncertainty estimation , which quantifies the reliability of each PI label ( Figure 1 ( b ) ) . Specifically , we propose to exploit a separate GNN only trained with the PI learning objective and calculate a confidence mask for each node pair from its PI prediction . Then , we filter out uncertain pairs by weighting the PI loss from another GNN jointly trained with the PI loss and node classification loss . Figure 1 ( c ) shows the accuracy of GNNs trained on the node pairs with higher and lower confidence on CiteSeer ( Yang et al. , 2016 ) and the former one performs better , which justifies the intuition of PI-GNN . Our main contributions are summarized as follows : • We propose robust GNNs against noisy labels for node classification , which serve as a crucial step towards the reliable deployment of GNNs in complex real-world applications . • We introduce a promising strategy to explicitly model the pairwise interactions and an uncertainty estimation approach to mitigate the negative effects of sub-optimal PI labels . • We demonstrate PI-GNN can be effectively applied on different datasets , GNN architectures and different noise types and rates , e.g. , improving the test accuracy by 6.7 % on CiteSeer with 80 % asymmetric noise . 2 RELATED WORK . Graph neural networks . Graph neural networks have been widely used to model the graphstructured data with various architectures , such as graph convolutional network ( GCN ) ( Kipf & Welling , 2017 ) , graph attention network ( GAT ) ( Velickovic et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) , Graph Isomorphism Network ( GIN ) ( Xu et al. , 2019 ) , Simple Graph Convolution ( SGC ) ( Wu et al. , 2019 ) , etc . Common graph analysis tasks , including node classification ( Park & Neville , 2019 ; Oono & Suzuki , 2020 ) , link prediction ( Baek et al. , 2020 ; Zhang & Chen , 2018 ) , graph classification ( Bacciu et al. , 2018 ; Errica et al. , 2020 ) , graph generation ( Liao et al. , 2019 ; Shi et al. , 2020 ) , have been widely studied in literature . However , only a few works focused on training robust GNNs against noisy labels , such as by loss correction ( NT et al. , 2019 ) for graph classification , sample re-weighting ( Xia et al. , 2021 ; Li et al. , 2021 ) for node classification . None of them exploited explicit pairwise interactions , which are compared with our PI-GNN in Section 5.4 . Bui et al . ( 2017 ) ; Stretcu et al . ( 2019 ) utilized graph structures for semi-supervised learning but with clean labels . Neural networks with noisy labels . Methods for neural networks against noisy labels can be roughly categorized into three types ( Han et al. , 2020b ) , i.e. , approaches from the perspective of data ( van Rooyen & Williamson , 2017 ) , objective ( Reed et al. , 2015 ; Miyato et al. , 2019 ) and optimization ( Arpit et al. , 2017 ) . Methods based on data mainly built the noise transition matrix to explore the data relationship between clean and noisy label by an adaptation layer ( Sukhbaatar et al. , 2015 ) , loss correction ( Patrini et al. , 2017 ) and prior knowledge ( Han et al. , 2018a ) . Methods based on objective modified the learning objective by regularization ( Han et al. , 2020a ) , reweighting ( Liu & Tao , 2016 ; Wang et al. , 2017 ) and loss redesign ( Thulasidasan et al. , 2019 ) . Methods based on optimization mainly changed the optimization policy , such as by memorization effect ( Jiang et al. , 2018 ) , self-training ( Ren et al. , 2018 ) and co-training ( Yu et al. , 2019 ) . Wu et al . ( 2020 ) proposed to use the similarity loss for noisy labels on image data but it relied on the noisy transition matrix , which is sensitive to the matrix estimation quality and can not use the graph structure for regularization . In this paper , we extend several approaches from each category to compare with PI-GNN in Section 5.4 . 3 PRELIMINARY . Graph Neural Networks . Let G = ( V , E ) be a graph with node feature vectors Xv for v ∈ V and edge set E. GNNs use the graph structure and node features Xv to learn a representation vector of a node hv , or the entire graph hG , which usually follow a neighborhood aggregation strategy and iteratively update the representation of a node by aggregating representations of its neighbors . After k iterations of aggregation , a node ’ s representation captures the structural information within its k-hop network neighborhood . Formally , the k-th layer of a GNN is a ( k ) v = AGGREGATE ( k ) ( { h ( k−1 ) u : u ∈N ( v ) } ) , h ( k ) v = COMBINE ( k ) ( h ( k−1 ) v , a ( k ) v ) , ( 1 ) where h ( k ) v is the feature vector of node v at the k-th layer . h ( 0 ) v = Xv . N ( v ) denotes the neighboring nodes of v. The choices of AGGREGATE ( k ) ( · ) and COMBINE ( k ) ( · ) can be diverse among different GNNs . For example , in GCN , the element-wise mean pooling is used , and the AGGREGATE and COMBINE steps are integrated as follows : h ( k ) v = ReLU ( W ·MEAN { h ( k−1 ) u , ∀u ∈N ( v ) ∪ { v } } ) , ( 2 ) where W is a learnable matrix . For node classification , each node v ∈V has an associated label yv , the node representation h ( K ) v of the final layer is used for prediction . Label-noise representation learning for GNNs . Let Xv be the feature and yv be the label for node v , we deal with a dataset D = { Dtr , Dte } which consists of training set Dtr = { ( A , Xv , yv ) } v∈V that is drawn from a corrupted distribution D = p ( A , X , Y ) where Y denotes the corrupted label . Let p ( A , X , Y ) be the non-corrupted joint probability distribution of features X and labels y , and f ∗ be the ( Bayes ) optimal hypothesis from X to y . To approximate f ∗ , the objective requires a hypothesis spaceH of hypotheses fθ ( · ) parametrized by θ . A robust algorithm against noisy labels contains the optimization policy to search throughH in order to find θ ∗ that corresponds to the optimal function in the hypothesis for Dtr : fθ∗ ∈H , and meanwhile is able to assign correct labels for Dte .

Summary: This paper proposes the PI-GNN (Pairwise Interactions in Graph Neural Network) model for semi-supervised tasks for nodes with noisy labels. It employs a new auxiliary task to predict PI (pairwise interactions) labels using a pair of nodes. Since it is costly to get clean PI labels, the authors use (1) two instantiations of PI labels: adjacency matrix and class agreement after label propagation and (2) uncertainty-aware (pre-)training with two model branches. The experiments on five real-world benchmarks demonstrate that PI-GNN outperforms GNN models (GCN, GraphSAGE, and GAT) and methods for noisy label settings (Decoupling, GCE, DivideMix, and so on.) on various noise levels and settings. PI label learning and uncertainty-aware training both contribute to performance improvement.