PI-GNN: Towards Robust Semi-Supervised Node Classification against Noisy Labels

1 INTRODUCTION . Graphs are ubiquitously used to represent data in different fields , including social networks , bioinformatics , recommendation systems , and computer network security . Accordingly , graph analysis tasks , such as node classification , have a significant impact in reality ( Lan et al. , 2020 ) . The success of machine learning models , such as graph neural networks ( GNNs ) on node classification relies heavily on the collection of large datasets with human-annotated labels ( Zhou et al. , 2019 ) . However , it is extremely expensive and time-consuming to label millions of nodes with high-quality annotations . Therefore , when dealing with large graphs , usually a subset of nodes is labeled , and a wide spectrum of semi-supervised learning techniques have emerged for improving node classification performance ( Zhu et al. , 2003 ; Zhou et al. , 2003 ; Kipf & Welling , 2017 ) . Although achieving promising results , these techniques overlook the existence of noisy node labels . For instance , practitioners often leverage inexpensive alternatives for annotation , such as combining human and machine-generated label ( Hu et al. , 2020 ) , which inevitably yields samples with noisy labels . Since neural networks ( including GNNs ) are able to memorize any given ( random ) labels ( NT et al. , 2019 ; Zhang et al. , 2017 ) , these noisy labels would easily prevent them from generalizing well . Therefore , training robust GNNs for semi-supervised node classification against noisy labels becomes increasingly crucial for safety-critical graph analysis , such as predicting the identity groups of users in social networks or the function of proteins to facilitate wet laboratory experiments , etc . A natural solution is extending the robust training techniques specifically designed for image data into graphs , such as by estimating noise transition matrix ( NT et al. , 2019 ) , re-weighting training samples ( Xia et al. , 2021 ; Li et al. , 2021 ; Dong et al. , 2020 ) and modifying model structure ( Li et al. , 2020a ) , which achieve promising results . However , there are two limitations . Firstly , some techniques require a clean set of node and label pairs for training , which are not practical to obtain under extremely noisy settings ( Li et al. , 2021 ) . Secondly , all of them solely rely on the noisy pointwise input-label interactions during training . Although some of them implicitly exploit other attributes ( the clean graph structure , etc . ) that encodes the pairwise information between nodes , such as aggregating features by GNNs on the input graphs or launching label propagation to get the pseudo labels for unlabeled nodes , they still operate on the noisy embeddings for inference , which may lead to limited performance . Thus , designing a robust GNN training technique for node classification against noisy labels is still a challenging problem . In this paper , we propose a novel framework , which explicitly leverages the Pairwise Interactions in Graph Neural Network ( PI-GNN ) to perform robust training without auxiliary clean labels . The PI , which is based on the similarity of two nodes , is meaningful for GNN against noisy labels because the noise rate for the PI labels can be much lower than that of the pointwise noisy class labels ( Figure 1 ( a ) ) . Consider two nodes from the same class have the same noisy labels , their similarity is still high w.r.t . the PI , which is helpful for the model to learn . Meanwhile , the pointwise information will inject both of these noisy class labels into GNN , which hurts its generalization performance . To ensure an effective PI learning procedure , two important components are highlighted . The first one is how to design informative PI labels . We propose two instantiations : 1 ) we use the adjacency matrix as PI labels so that if two nodes are connected , their PI label is positive . 2 ) We compare the noisy labels , i.e. , two nodes that have the same class labels are given a positive PI label . During training , PI-GNN explicitly forces the embedding similarity of two nodes to be close to the PI labels . The second one is how to deal with the sub-optimal PI labels during regularization . It is critical to notice that the two instantiations are not optimal . Firstly , connected nodes do not guarantee to have the same ( clean ) class label , so their PI label is not always positive . Secondly , the class labels are noisy , and comparing them to obtain the PI labels inevitably introduces noise . To address this problem , we introduce uncertainty estimation , which quantifies the reliability of each PI label ( Figure 1 ( b ) ) . Specifically , we propose to exploit a separate GNN only trained with the PI learning objective and calculate a confidence mask for each node pair from its PI prediction . Then , we filter out uncertain pairs by weighting the PI loss from another GNN jointly trained with the PI loss and node classification loss . Figure 1 ( c ) shows the accuracy of GNNs trained on the node pairs with higher and lower confidence on CiteSeer ( Yang et al. , 2016 ) and the former one performs better , which justifies the intuition of PI-GNN . Our main contributions are summarized as follows : • We propose robust GNNs against noisy labels for node classification , which serve as a crucial step towards the reliable deployment of GNNs in complex real-world applications . • We introduce a promising strategy to explicitly model the pairwise interactions and an uncertainty estimation approach to mitigate the negative effects of sub-optimal PI labels . • We demonstrate PI-GNN can be effectively applied on different datasets , GNN architectures and different noise types and rates , e.g. , improving the test accuracy by 6.7 % on CiteSeer with 80 % asymmetric noise . 2 RELATED WORK . Graph neural networks . Graph neural networks have been widely used to model the graphstructured data with various architectures , such as graph convolutional network ( GCN ) ( Kipf & Welling , 2017 ) , graph attention network ( GAT ) ( Velickovic et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) , Graph Isomorphism Network ( GIN ) ( Xu et al. , 2019 ) , Simple Graph Convolution ( SGC ) ( Wu et al. , 2019 ) , etc . Common graph analysis tasks , including node classification ( Park & Neville , 2019 ; Oono & Suzuki , 2020 ) , link prediction ( Baek et al. , 2020 ; Zhang & Chen , 2018 ) , graph classification ( Bacciu et al. , 2018 ; Errica et al. , 2020 ) , graph generation ( Liao et al. , 2019 ; Shi et al. , 2020 ) , have been widely studied in literature . However , only a few works focused on training robust GNNs against noisy labels , such as by loss correction ( NT et al. , 2019 ) for graph classification , sample re-weighting ( Xia et al. , 2021 ; Li et al. , 2021 ) for node classification . None of them exploited explicit pairwise interactions , which are compared with our PI-GNN in Section 5.4 . Bui et al . ( 2017 ) ; Stretcu et al . ( 2019 ) utilized graph structures for semi-supervised learning but with clean labels . Neural networks with noisy labels . Methods for neural networks against noisy labels can be roughly categorized into three types ( Han et al. , 2020b ) , i.e. , approaches from the perspective of data ( van Rooyen & Williamson , 2017 ) , objective ( Reed et al. , 2015 ; Miyato et al. , 2019 ) and optimization ( Arpit et al. , 2017 ) . Methods based on data mainly built the noise transition matrix to explore the data relationship between clean and noisy label by an adaptation layer ( Sukhbaatar et al. , 2015 ) , loss correction ( Patrini et al. , 2017 ) and prior knowledge ( Han et al. , 2018a ) . Methods based on objective modified the learning objective by regularization ( Han et al. , 2020a ) , reweighting ( Liu & Tao , 2016 ; Wang et al. , 2017 ) and loss redesign ( Thulasidasan et al. , 2019 ) . Methods based on optimization mainly changed the optimization policy , such as by memorization effect ( Jiang et al. , 2018 ) , self-training ( Ren et al. , 2018 ) and co-training ( Yu et al. , 2019 ) . Wu et al . ( 2020 ) proposed to use the similarity loss for noisy labels on image data but it relied on the noisy transition matrix , which is sensitive to the matrix estimation quality and can not use the graph structure for regularization . In this paper , we extend several approaches from each category to compare with PI-GNN in Section 5.4 . 3 PRELIMINARY . Graph Neural Networks . Let G = ( V , E ) be a graph with node feature vectors Xv for v ∈ V and edge set E. GNNs use the graph structure and node features Xv to learn a representation vector of a node hv , or the entire graph hG , which usually follow a neighborhood aggregation strategy and iteratively update the representation of a node by aggregating representations of its neighbors . After k iterations of aggregation , a node ’ s representation captures the structural information within its k-hop network neighborhood . Formally , the k-th layer of a GNN is a ( k ) v = AGGREGATE ( k ) ( { h ( k−1 ) u : u ∈N ( v ) } ) , h ( k ) v = COMBINE ( k ) ( h ( k−1 ) v , a ( k ) v ) , ( 1 ) where h ( k ) v is the feature vector of node v at the k-th layer . h ( 0 ) v = Xv . N ( v ) denotes the neighboring nodes of v. The choices of AGGREGATE ( k ) ( · ) and COMBINE ( k ) ( · ) can be diverse among different GNNs . For example , in GCN , the element-wise mean pooling is used , and the AGGREGATE and COMBINE steps are integrated as follows : h ( k ) v = ReLU ( W ·MEAN { h ( k−1 ) u , ∀u ∈N ( v ) ∪ { v } } ) , ( 2 ) where W is a learnable matrix . For node classification , each node v ∈V has an associated label yv , the node representation h ( K ) v of the final layer is used for prediction . Label-noise representation learning for GNNs . Let Xv be the feature and yv be the label for node v , we deal with a dataset D = { Dtr , Dte } which consists of training set Dtr = { ( A , Xv , yv ) } v∈V that is drawn from a corrupted distribution D = p ( A , X , Y ) where Y denotes the corrupted label . Let p ( A , X , Y ) be the non-corrupted joint probability distribution of features X and labels y , and f ∗ be the ( Bayes ) optimal hypothesis from X to y . To approximate f ∗ , the objective requires a hypothesis spaceH of hypotheses fθ ( · ) parametrized by θ . A robust algorithm against noisy labels contains the optimization policy to search throughH in order to find θ ∗ that corresponds to the optimal function in the hypothesis for Dtr : fθ∗ ∈H , and meanwhile is able to assign correct labels for Dte .

Summary: This paper focuses on semi-supervised node classification with noisy node labels. The authors propose a novel learning objective called the pairwise interactions, which encourages node pairs holding positive PI labels to have close node embeddings. Extensive experiments show promising results.